{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Goal in brief ###\n",
    "\n",
    "Registrating brain images (2D/3D) with NNs. \n",
    "\n",
    "### Survey of architectures / Possible ideas ###\n",
    "Most recent work are all using a \"bottleneck\" structure somewhere.  \n",
    "* <span style =\"color: blue\"> Add a VAE infront for generalization purpose? It also provides a way to compress the data</span>\n",
    "\n",
    "Some work utilizes `spatial transformation netwrok` to enforce the net learns spatial transforms.\n",
    "* <span style =\"color: blue\"> Use certain loss (e.g. gradient related, if so, apply a gaussian filter first?) to regularize? </span>\n",
    "\n",
    "* Assuming two templates are already proper aligned. If not, could stack a spatial transformer layer ahead for learning the proper affine (or other) transformations. \n",
    "\n",
    "\n",
    "### Other Ideas ###\n",
    "* A two stage registration. First stage learns the spatial relationship, second stage learns to draw the intensity better.        \n",
    "   <span style = 'color: blue'> How to decide if the spatial relationship is learned adequetly? using certain kinds of contour   mask?\n",
    "   \n",
    "* Taking the idea of GAN, let a discriminator to justify the quality from the output image. Using ANTs or similar to provide warped \"ground truth\". (What is the merit of this? Can it provide better result than the purely unsupervised way?)\n",
    "\n",
    "### Questions during actual implementation###\n",
    "* In the code `spatialdeformer.py`, why \n",
    "```\n",
    "deformation = tf.reshape(deformation, shape = (batch_size, 2, -1) )\n",
    "```\n",
    "\n",
    "is not the same as \n",
    "\n",
    "```\n",
    "deformation = tf.reshape(deformation, (-1, output_height * output_width, 2))\n",
    "deformation = tf.transpose(deformation, (0, 2, 1))\n",
    "```\n",
    "\n",
    "The latter gives desired result, the former does not (Gradient seems to be very small for its update).\n",
    "\n",
    "<span style = 'color: blue'> The truth is the mechanism of `tf.reshape()`, it will start filling the tensor using digits beginning from the last dimension of the original tensor. The deformation is of shape (batch_size, height, width, 2). So in order to make the information stay in the same channel, you have to do it the latter way. Even though these weight are latent and learned during training.</span>\n",
    "\n",
    "* Stacking a STN infront of a SDN will produce a warning relating to converting a sparse tense into a dense one implicitly.\n",
    "(The out put of STN is of shape (?, height, width, ?), why the channel info is not seen?)\n",
    "\n",
    "* Is it necessary to do some preprocess like substract mean from trainning channel-wised?\n",
    "\n",
    "<span style = 'color: blue'> Does not seem to help much, there will be stripes at the border, and the loss curve does not look nice. Maybe other preprocess will help, check what do others do. </span>\n",
    "\n",
    "\n",
    "* Deformation, when as an extra output, returns NAN when using total variation loss.\n",
    "    <span style = 'color: blue'> Solved by initializing the displacement layer with NONZEROS </span>\n",
    "    \n",
    "* Need a module for visualization/exploration purposes.\n",
    "   * Check plotly, mayavi\n",
    "   * plt.axes3d.voxels method seems to be very slow.\n",
    "   * Check [pysufer](https://pysurfer.github.io/install.html)\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Focused on losses###\n",
    "\n",
    "* Single loss: means_sqaure_logarithmic_error and KL-divergence seems to work good\n",
    "   \n",
    "     * Their scales on train. KL: (negative when using img directly, ~0.10 when flattened and normalized by sum)\n",
    "                              SobelLoss: ~0.08 \n",
    "                              MSE: ~0.08\n",
    "                              MSE_log: ~0.005   \n",
    "                              BCE: ~0.3\n",
    "                              TVS: ~0.02 with $\\alpha = 1.25$\n",
    "\n",
    "* Using perception loss extracted from other pretrained models.\n",
    "\n",
    "   <span style=\"color: blue\"> This may be a general solution if one want to combine different features but is not sure what particular kinds of feature should be used manually.</span>\n",
    "   \n",
    "   Perceptual losses from pretrained VGG16 net on imagenet. \n",
    "   * MSE loss from a single layer. Using only block3 will have \"water rippples\" effect comapred to block4\n",
    "   * MSE loss from multiple layer. Scales of different losses roughly are : 0.5, 10, 15, 0.5, 0.01.\n",
    "     <span style = 'color: blue'> How to balance different losses so as to better assign weights? </span>\n",
    "     \n",
    "   * \"Stylish\" losses that uses the gram matrix\n",
    "   * \"Correlation\" losses.\n",
    "\n",
    "* Current loss used: KL + SobelLoss + BCE\n",
    "\n",
    "* What about mutual information (MI): https://matthew-brett.github.io/teaching/mutual_information.html\n",
    "\n",
    "\n",
    "                              \n",
    "### Structure of the regress net ###\n",
    "\n",
    "* Choice of uppooling: upsampling or conv_transpose.\n",
    "\n",
    "* Adding paralelled SDN modules.\n",
    "\n",
    "* Adding some short-cut \"highway\"\n",
    " \n",
    "   **<span style = \"color: blue\"> A `multiply` layer between an eariler conv layer (detecting edges) and a latter one shows an impressive improvement in learning !! This may because feature maps produced at the earlier conv when mulitplied with the latter feature map help put attention on \"edges\", hence learns the shape better.</span>**\n",
    "   \n",
    "\n",
    "* Does not seems to be the case for OASIS brain slices, but in case of too much freedom on the deformation. A proper regularization loss should be considered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### TODO ###\n",
    "\n",
    "* Find a 2D dataset for evaluation: Jacard or landmark distance.\n",
    "     * http://www.bic.mni.mcgill.ca/ServicesAtlases/ICBM152NLin2009\n",
    "     * LONI\n",
    "       Need to learn how the data is organised and their formats.       \n",
    "            * current using simpleitk for access the analyse format. An \n",
    "            [example](http://insightsoftwareconsortium.github.io/SimpleITK-Notebooks/Python_html/10_matplotlib's_imshow.html)\n",
    "            * Images does not seem to be affine-aligned.Some of the heads are tilted. The ROI data is not very good (at least visually)\n",
    "* Utilreg or other to produce $P$ for adding a supervised learning module.\n",
    "* Extend the current framework to 3d.\n",
    "* How to write a regularization term?\n",
    "     * As one of the loss\n",
    "     * regularization method in layer class [DONE]\n",
    "* Sample/interpolate directly from images of the original resolution with generated grids of different resolution instead of presampling training images to desired resolution?\n",
    "   <span style = 'color:blue'> Need to reimplement SDN </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem Encountered ###\n",
    "\n",
    "* The deformation network deform the background as well. When there are relative shifts, the deformation does not seem to be very smooth. This is because the network does not learn to distinguish objects but tries to make changes on edges so that it \"looks like\" the target. What **regularization** will produce desired results? (Less emphasizing \"edges\"?)\n",
    "    * <span style = 'color:blue'> It seems that a fine tuned regularization strength will help. Use different weights for displacement and its derivatives.</span>\n",
    "\n",
    "\n",
    "* **Warping small ventricles to large ones seems to be more difficult than the other way around**. \n",
    "\n",
    "<span style = 'color: blue'> This may because of the nature of your SDN which uses the warped grid to sample, if the moving image has a gap that is only 2 or 3 pixels wide, it is hard for many grid points to be squeezed in that gap. </span>\n",
    "\n",
    "* It is hard to learn local sturctures.\n",
    "\n",
    "   * Idea one: Use certain loss function wrt `shapes`: countours...\n",
    "   * Idea two: How to enforce the **\"edge to edge\"** relationship with a mask of the target image.\n",
    "   \n",
    "        Multiply the mask directly (or its exponential)to the predicted image/displacement to \"boost\" values on the edges?\n",
    "   * Idea three: Maybe a correlation feature map like in Rocco's paper will help?\n",
    "\n",
    "* AntsR imageSimilarity does strange things...\n",
    "* *Need to check if background is 0 before feed them to NN*!\n",
    "* Am I doing the right thing for visualizing the warped grid. The value plot does not seem to be consistent with the grid plot... what do others do?\n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Links ###\n",
    "\n",
    "* Software here seems to be very [technical](http://resource.loni.usc.edu/resources/downloads)\n",
    "* Get to know the field: \n",
    "   * some popular algorithms: DARTEL, geodesic shooting, diffeo-demons, SyN (symmetric normalization), LDDMM "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
